Skip to content

feat(integration): migrate MCP client to use direct API server calls#667

Closed
naren-venhan wants to merge 1 commit intocoleam00:mainfrom
sadhakaai:feat/azure-cloud-e-commerce
Closed

feat(integration): migrate MCP client to use direct API server calls#667
naren-venhan wants to merge 1 commit intocoleam00:mainfrom
sadhakaai:feat/azure-cloud-e-commerce

Conversation

@naren-venhan
Copy link
Copy Markdown

@naren-venhan naren-venhan commented Sep 15, 2025

  • Replace streamable HTTP MCP protocol calls with direct REST API calls
  • Implement all MCP tool methods to call respective API server endpoints
  • Maintain backward compatibility with MCP tool interface
  • Add detailed error handling and logging for HTTP and unexpected errors
  • Update environment variable usage for MCP server URLs and backend service URLs
  • Enhance proxy and Vite config to support Azure Container Apps URLs
  • Refactor MCP server to include health check endpoints for external monitoring
  • Register new E-commerce and Specialized Crawling modules as HTTP-based MCP tools
  • Improve project data fetching to include associated docs and features on selection
  • Add support for switching tabs with data refreshing in ProjectPage component
  • Upgrade DocsTab component to handle absence of docs field gracefully and log info
  • Add specialized crawling API endpoints and crawling modes listing for enhanced crawling features
  • Revise Dockerfile for multi-stage build with production nginx serving and environment variables
  • Adjust document restoration API call to include additional body parameters for audit purposes

Pull Request

Summary

Changes Made

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Performance improvement
  • Code refactoring

Affected Services

  • Frontend (React UI)
  • Server (FastAPI backend)
  • MCP Server (Model Context Protocol)
  • Agents (PydanticAI service)
  • Database (migrations/schema)
  • Docker/Infrastructure
  • Documentation site

Testing

  • All existing tests pass
  • Added new tests for new functionality
  • Manually tested affected user flows
  • Docker builds succeed for all services

Test Evidence

# Example: python -m pytest tests/
# Example: cd archon-ui-main && npm run test

Checklist

  • My code follows the service architecture patterns
  • If using an AI coding assistant, I used the CLAUDE.md rules
  • I have added tests that prove my fix/feature works
  • All new and existing tests pass locally
  • My changes generate no new warnings
  • I have updated relevant documentation
  • I have verified no regressions in existing features

Breaking Changes

Additional Notes

Summary by CodeRabbit

  • New Features

    • Smart Crawling system with website-type detection, configurable modes (e-commerce/blog/docs/analytics), mode performance, and price intelligence APIs.
    • New UI panel to view/edit crawling mode configs.
    • MCP health endpoints and cloud (Azure) support with improved service discovery.
    • Frontend now served via Nginx with runtime-configurable backend URLs.
  • Bug Fixes

    • Fixed Archon UI port configuration and improved Docs tab reliability.
    • More robust MCP client URL resolution and frontend MCP integration.
  • Documentation

    • Added guides for MCP server fixes, Smart Crawling, and Specialized Crawling.
  • Chores

    • New Azure deployment/redeploy scripts and environment presets.
  • Tests

    • Comprehensive e-commerce crawling test suite and runners.

- Replace streamable HTTP MCP protocol calls with direct REST API calls
- Implement all MCP tool methods to call respective API server endpoints
- Maintain backward compatibility with MCP tool interface
- Add detailed error handling and logging for HTTP and unexpected errors
- Update environment variable usage for MCP server URLs and backend service URLs
- Enhance proxy and Vite config to support Azure Container Apps URLs
- Refactor MCP server to include health check endpoints for external monitoring
- Register new E-commerce and Specialized Crawling modules as HTTP-based MCP tools
- Improve project data fetching to include associated docs and features on selection
- Add support for switching tabs with data refreshing in ProjectPage component
- Upgrade DocsTab component to handle absence of docs field gracefully and log info
- Add specialized crawling API endpoints and crawling modes listing for enhanced crawling features
- Revise Dockerfile for multi-stage build with production nginx serving and environment variables
- Adjust document restoration API call to include additional body parameters for audit purposes
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Sep 15, 2025

Caution

Review failed

The pull request is closed.

Walkthrough

Introduces Azure-focused deployment and configuration scripts, overhauls Archon UI build/runtime (Vite/nginx) and env handling, adds Smart Crawling frameworks (modes, detection, pagination, price intelligence), expands server APIs (smart crawl, knowledge, MCP cloud mode), updates service discovery for Azure, adds MCP modules/tools, modifies frontend components/services, and adds extensive SQL migrations and tests.

Changes

Cohort / File(s) Summary
Deployment & Fix Scripts
archon-azure-setup.ps1, FixArchonUIPortConfiguration.ps1, FixFrontendBackendURLs.ps1, Rebuild-ArchonUI.ps1, Rebuild-ArchonUI-Env.ps1, fix-frontend-mcp.ps1, fix-mcp-server.ps1, quick-fix-mcp.ps1, redeploy-archon-ui-url-whitelist.sh
New scripts to deploy/update Azure Container Apps, rebuild/push Docker images, set env vars, fix ports (5173), wire frontend-backend URLs, validate MCP endpoints.
Frontend Build/Runtime Config
archon-ui-main/Dockerfile, archon-ui-main/entrypoint.sh, archon-ui-main/nginx.conf.template, archon-ui-main/azure.env, archon-ui-main/vite.config.js, archon-ui-main/vite.config.ts
Switch to multi-stage build with nginx runtime and templated config; add entrypoint/env substitution; add Azure env file; unify dev/preview at 5173; proxy based on VITE_API_URL and VITE_MCP_SERVER_URL; remove host/port env exports.
Frontend Components/Services
archon-ui-main/src/components/project-tasks/DocsTab.tsx, archon-ui-main/src/components/smart-crawling/SmartCrawlConfig.tsx, archon-ui-main/src/pages/ProjectPage.tsx, archon-ui-main/src/services/mcpClientService.ts, archon-ui-main/src/services/projectService.ts
DocsTab null-safety and logging; add SmartCrawlConfig panel (load/edit/save modes); ProjectPage fetches full project on select/tab/pin; MCP client resolves Azure MCP or local; restoreDocumentVersion sends JSON body.
Server APIs
python/src/server/main.py, python/src/server/api_routes/smart_crawl_api.py, python/src/server/api_routes/knowledge_api.py, python/src/server/api_routes/mcp_api.py
Register Smart Crawl router; add smart crawling endpoints (crawl, modes, config, ecommerce, analytics); extend knowledge API for specialized crawl and mode perf; MCP API gains cloud vs docker mode, health monitor, force-health-check, and HTTP tool fetching.
Service Discovery & Clients
python/src/server/config/service_discovery.py, python/src/server/services/mcp_service_client.py, python/src/agents/mcp_client.py
Add Azure environment detection/URLs; environment-aware timeouts; improved health logs; agents MCP client switches to REST/HTTP paths and API URL resolution.
MCP Server & Modules
python/src/mcp/mcp_server.py, python/src/mcp/modules/ecommerce_module.py, python/src/mcp/modules/specialized_crawling_module.py
Register ecommerce/specialized modules; add MCP health tools; modules call server HTTP APIs for crawl, search, modes, details.
Crawling Core: Modes/Detection/Orchestration
python/src/server/services/crawling/modes/base.py, .../base_mode.py, .../standard_mode.py, .../ecommerce.py, .../ecommerce_mode.py, .../blog.py, .../documentation.py, .../analytics.py, .../mode_registry.py, .../registry.py, .../config_manager.py, .../detection/__init__.py, .../detection/website_detector.py, .../smart_orchestrator.py, .../pagination_handler.py, .../price_intelligence.py, python/src/server/services/crawling/crawling_service.py
Introduce extensible crawling framework: base contracts, multiple modes, intelligent detection, registry, config manager (Supabase-backed), smart orchestrator with progress/session storage, pagination handling, price intelligence, and service integration; update CrawlingService to use specialized modes.
Database Migrations
migration/add_specialized_crawling_tables.sql, migration/verify_specialized_crawling.sql, migration/smart_crawling_modes_schema.sql
Add specialized crawling schemas (products, variants, price history, reviews, modes, sessions, structured data), views, triggers, RLS policies; insert defaults; verification script for post-migration checks.
Tests & Utilities
python/tests/*, python/test_mcp_config.py, python/tests/test_requirements.txt, python/tests/README.md
Add comprehensive tests for crawling modes, detection, pagination, stealth, pricing; test runner/reporting; mocks and fixtures; MCP config test; testing requirements and docs.
Docs
MCP-SERVER-FIX-README.md, SMART_CRAWLING_GUIDE.md, SPECIALIZED_CRAWLING.md
New documentation for MCP fixes and Smart/Specialized Crawling architecture, usage, and configuration.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant UI as Frontend (archon-ui)
  participant API as Archon API
  participant Orchestrator as SmartCrawlingOrchestrator
  participant Registry as ModeRegistry
  participant Mode as Mode Instance
  participant DB as Supabase

  UI->>API: POST /api/smart-crawl/crawl (urls, source_id, [force_mode])
  activate API
  API->>Orchestrator: smart_crawl(urls, source_id, cb)
  activate Orchestrator
  Orchestrator->>Registry: detect_best_mode(url) for each
  Registry-->>Orchestrator: mode assignments
  Orchestrator->>Registry: create/get mode instances
  loop per mode
    Orchestrator->>Mode: crawl(url) with progress callbacks
    Mode-->>Orchestrator: ExtractedData[]
  end
  Orchestrator->>DB: Store ecommerce/products, variants, sessions
  Orchestrator-->>API: CrawlResult summary (ids, stats)
  deactivate Orchestrator
  API-->>UI: 202 Accepted {progress_id}
  deactivate API
  note over UI,API: UI polls socket/progress or endpoints for updates
Loading
sequenceDiagram
  autonumber
  participant UI as Frontend
  participant API as MCP API Router
  participant Disc as ServiceDiscovery
  participant MCP as MCP Server (HTTP)
  participant Docker as Docker (optional)

  API->>Disc: detect environment
  alt Cloud (Azure)
    API->>MCP: GET /health (HTTP)
    MCP-->>API: 200/starting
    API-->>UI: status {mode: cloud, health}
    UI->>API: POST /api/mcp/force-health-check
    API->>MCP: GET /health (forced)
    MCP-->>API: 200
    API-->>UI: {healthy: true}
  else Docker
    API->>Docker: start/stop/status via container runtime
    Docker-->>API: container state
    API-->>UI: status {mode: docker, state}
  end
Loading

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120+ minutes

Possibly related PRs

Suggested labels

enhancement

Suggested reviewers

  • leex279
  • Wirasm
  • coleam00

Poem

Hop, hop—new modes I bring,
Azure skies where proxies sing.
Ports aligned at five-one-seven-three,
Nginx hums its static spree.
Smart paws crawl, detect, and write—
Products priced by lunar light.
Ship it swift—ears up, delight! 🐇✨

✨ Finishing touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ee3af43 and 39314a8.

📒 Files selected for processing (64)
  • FixArchonUIPortConfiguration.ps1 (1 hunks)
  • FixFrontendBackendURLs.ps1 (1 hunks)
  • MCP-SERVER-FIX-README.md (1 hunks)
  • Rebuild-ArchonUI-Env.ps1 (1 hunks)
  • Rebuild-ArchonUI.ps1 (1 hunks)
  • SMART_CRAWLING_GUIDE.md (1 hunks)
  • SPECIALIZED_CRAWLING.md (1 hunks)
  • archon-azure-setup.ps1 (1 hunks)
  • archon-ui-main/Dockerfile (1 hunks)
  • archon-ui-main/azure.env (1 hunks)
  • archon-ui-main/entrypoint.sh (1 hunks)
  • archon-ui-main/nginx.conf.template (1 hunks)
  • archon-ui-main/src/components/project-tasks/DocsTab.tsx (3 hunks)
  • archon-ui-main/src/components/smart-crawling/SmartCrawlConfig.tsx (1 hunks)
  • archon-ui-main/src/pages/ProjectPage.tsx (3 hunks)
  • archon-ui-main/src/services/mcpClientService.ts (1 hunks)
  • archon-ui-main/src/services/projectService.ts (1 hunks)
  • archon-ui-main/vite.config.js (1 hunks)
  • archon-ui-main/vite.config.ts (4 hunks)
  • fix-frontend-mcp.ps1 (1 hunks)
  • fix-mcp-server.ps1 (1 hunks)
  • migration/add_specialized_crawling_tables.sql (1 hunks)
  • migration/smart_crawling_modes_schema.sql (1 hunks)
  • migration/verify_specialized_crawling.sql (1 hunks)
  • python/src/agents/mcp_client.py (4 hunks)
  • python/src/mcp/mcp_server.py (2 hunks)
  • python/src/mcp/modules/ecommerce_module.py (1 hunks)
  • python/src/mcp/modules/specialized_crawling_module.py (1 hunks)
  • python/src/server/api_routes/knowledge_api.py (2 hunks)
  • python/src/server/api_routes/mcp_api.py (13 hunks)
  • python/src/server/api_routes/smart_crawl_api.py (1 hunks)
  • python/src/server/config/service_discovery.py (8 hunks)
  • python/src/server/main.py (2 hunks)
  • python/src/server/services/crawling/crawling_service.py (3 hunks)
  • python/src/server/services/crawling/detection/__init__.py (1 hunks)
  • python/src/server/services/crawling/detection/website_detector.py (1 hunks)
  • python/src/server/services/crawling/modes/__init__.py (1 hunks)
  • python/src/server/services/crawling/modes/analytics.py (1 hunks)
  • python/src/server/services/crawling/modes/base.py (1 hunks)
  • python/src/server/services/crawling/modes/base_mode.py (1 hunks)
  • python/src/server/services/crawling/modes/blog.py (1 hunks)
  • python/src/server/services/crawling/modes/config_manager.py (1 hunks)
  • python/src/server/services/crawling/modes/detector.py (1 hunks)
  • python/src/server/services/crawling/modes/documentation.py (1 hunks)
  • python/src/server/services/crawling/modes/ecommerce.py (1 hunks)
  • python/src/server/services/crawling/modes/ecommerce_mode.py (1 hunks)
  • python/src/server/services/crawling/modes/mode_registry.py (1 hunks)
  • python/src/server/services/crawling/modes/registry.py (1 hunks)
  • python/src/server/services/crawling/modes/standard_mode.py (1 hunks)
  • python/src/server/services/crawling/pagination_handler.py (1 hunks)
  • python/src/server/services/crawling/price_intelligence.py (1 hunks)
  • python/src/server/services/crawling/smart_orchestrator.py (1 hunks)
  • python/src/server/services/crawling/stealth_crawler.py (1 hunks)
  • python/src/server/services/crawling/testing/__init__.py (1 hunks)
  • python/src/server/services/crawling/testing/crawling_tests.py (1 hunks)
  • python/src/server/services/mcp_service_client.py (8 hunks)
  • python/test_mcp_config.py (1 hunks)
  • python/tests/README.md (1 hunks)
  • python/tests/conftest.py (1 hunks)
  • python/tests/run_tests.py (1 hunks)
  • python/tests/test_ecommerce_crawling.py (1 hunks)
  • python/tests/test_requirements.txt (1 hunks)
  • quick-fix-mcp.ps1 (1 hunks)
  • redeploy-archon-ui-url-whitelist.sh (1 hunks)

Tip

👮 Agentic pre-merge checks are now available in preview!

Pro plan users can now enable pre-merge checks in their settings to enforce checklists before merging PRs.

  • Built-in checks – Quickly apply ready-made checks to enforce title conventions, require pull request descriptions that follow templates, validate linked issues for compliance, and more.
  • Custom agentic checks – Define your own rules using CodeRabbit’s advanced agentic capabilities to enforce organization-specific policies and workflows. For example, you can instruct CodeRabbit’s agent to verify that API documentation is updated whenever API schema files are modified in a PR. Note: Upto 5 custom checks are currently allowed during the preview period. Pricing for this feature will be announced in a few weeks.

Please see the documentation for more information.

Example:

reviews:
  pre_merge_checks:
    custom_checks:
      - name: "Undocumented Breaking Changes"
        mode: "warning"
        instructions: |
          Pass/fail criteria: All breaking changes to public APIs, CLI flags, environment variables, configuration keys, database schemas, or HTTP/GraphQL endpoints must be documented in the "Breaking Change" section of the PR description and in CHANGELOG.md. Exclude purely internal or private changes (e.g., code not exported from package entry points or explicitly marked as internal).

Please share your feedback with us on this Discord post.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant